1. Field of the Invention
The present invention relates to an apparatus for reconstructing an image of an arbitrary view point (or an arbitrary point of view (POV)) from plural photographed images.
2. Related Background Art
There have been attempts to generate an image of an arbitrary view point from photographed images. For example there is employed a method of determining a shape of an object from two or more images utilizing a stereo measurement, and generating an image of an arbitrary view point by a computer graphics (CG) technology. In such method, however, it is necessary to identify corresponding points in two or more images, and such operation is very difficult under certain conditions thereby resulting in a low image quality of a generated image. In contrast, an image-based rendering (IBR) technology, particularly a method of obtaining a light beam coming into the space and generating an image of an arbitrary view point utilizing such beam, has an advantage capable in principle of generating a proper image of an arbitrary view point by measuring a position and an orientation (or posture) of a camera at an image pickup (image being a record of a ray passing through a lens of the camera).
FIG. 31A is a schematic view showing a CCD camera in a state of image pickup of an object. An ordinary image pickup plane is a CCD, but, for the simplicity of explanation, an imaginary plane folded back symmetrically with respect to a center of a lens is assumed as an image pickup plane, and, for further simplifying the explanation, the image pickup plane is assumed to be constituted of a one-dimensional line sensor. As shown in FIG. 31B, in case a multi view point image can be obtained at an arbitrary view point on a line L, a color of a pixel i of an image observable at a view point position Q is same as a color of a pixel i′ of an image having a view point position at a crossing point R of a line connecting the pixel i and the view point position Q and the line L. It is thus possible to reconstruct an image at a completely new view point position at which the image pickup has not been made, by determining such relationship for all the pixels of the image to be reconstructed. In “Light Field Rendering” (M. Levoy and P. Hanrahan, Proc. SIGGRAPH 96, p. 31-42, 1996), which is a representative example of such method, the light beams are acquired by photographing images with cameras positioned in lattice points as shown in FIG. 32.
In the aforementioned method, however, since image are taken in advance at plural view point positions for obtaining the light beams, then such images are once stored in a storage apparatus and are used for generating an image at an arbitrary view point, it is necessary to secure a storage area for storing plural images. Also since the number of the photographed images affects the image quality of the image of the arbitrary view point to be generated, it is necessary to take a large number of images for obtaining a high-quality image, so that a recording apparatus for storing the photographed images also requires an enormous capacity. Such situation may be tolerated in an equipment connectable to a recording apparatus of a large capacity such as a personal computer, but constitutes a major difficulty in an equipment such as a compact digital camera, since the storage capacity is limited.
Also in case of an image pickup apparatus such as a digital camera utilizing the image-based rendering technology, after an image reconstruction is started by a depression of a start button by the user, it is not possible to recognize a current situation or to understand when the process will be terminated, whereby the user may feel stressed.